Image sensing apparatus sensing moving and still images, method thereof, storage medium and computer program

ABSTRACT

An image sensing apparatus includes an imager, an image reducer for reducing an image size from imaging data obtained from the imager, a moving image configurer for rendering an image reduced by the image reducer as a moving image, an image processor for image-processing image data, a network communicator, and a storing device for storing frame image data obtained from the imager in response to a control instruction from the outside while taking a moving image. The image data stored in the storing device is image-processed by dividing it into non-operating periods of moving image processing per frame configuring the moving image. Also, parallel processing of a moving image and a still image is performed without lowering a frame rate of the moving image.

FIELD OF THE INVENTION

The present invention relates to an image sensing apparatus, a methodthereof, a storage medium and a computer program, and in particular, toa suitable technology for processing a moving image and a still image inparallel for instance.

BACKGROUND OF THE INVENTION

A conventional image sensing apparatus, which captures a high densitystill image when instructed to capture a still image by externalmanipulation while sensing a moving image, is provided. For example, theapparatus has multiple image processing systems, one for a still imageand another for a moving image, and processes a still image in a stillimage system. The still image to be processed is constructed withseveral frames by sequentially processing area portions of a screenseparated by a frame period or a field period, as in Japanese PatentLaid-Open No. 2001-326896.

For another example, an apparatus having an image sensing device forreading out moving and still images in separate modes is provided. Theapparatus can switch to still image reading on receipt of a still imagecontinuous sensing command while storing a moving image, to continuouslyshoot only a preset number of still images, and repeatedly processes onemoving image and one still image. It is arranged to lower the frame rateof moving image data to process the still image in this case, refer toJapanese Patent Laid-Open No. 2003-158653.

As for the conventional examples, however, Japanese Patent Laid-Open No.2001-326896 has problems that its LSI chip area increases because it isarranged to have multiple image processing blocks for the still imagesand for the moving images. Also, power consumption increases because itperforms simultaneous parallel image processing of the moving image andthe still image. As for Japanese Patent Laid-Open No. 2003-158653, thereis a problem that the frame rate of the moving image decreases oncapturing the still image while storing the moving image.

SUMMARY OF THE INVENTION

In view of the aforementioned problems, an object of the presentinvention is to allow parallel processing of a moving image and a stillimage without lowering the frame rate of the moving image.

A further object is to prevent the area of the image processing chipfrom increasing.

To solve these problems, the present invention provides an image sensingapparatus and the like comprising: imaging means; image reducing meansfor reducing an image size from imaging data obtained from the imagingmeans; moving image configuring means for rendering an image reduced bythe image reducing means as a moving image; image processing means forimage-processing image data; network communication means; and image datastoring means for storing frame image data obtained from the imagingmeans in response to a control instruction from outside while taking amoving image. The image data stored in the image data storing means isimage-processed by dividing it into non-operating periods of movingimage processing per frame configuring the moving image.

According to the present invention described above, it is possible, whencapturing a still image of a larger size than a moving image whiletaking the moving image, to deliver it on a network or record it withoutlowering the frame rate of the moving image at all.

Furthermore, continuous sensing of large-screen still images whiletaking the moving image becomes possible within the entire pixel memoryamount, and continuous sensing by VD becomes possible so as to obtainthe images continuously shot at the same speed as moving image frames.

Furthermore, it is not necessary to have multiple image processingcircuit blocks, and so the control can be realized with no need toincrease the circuit scale on rendering it as LSI. It is possible toreduce power consumption of circuits by preventing the increase in thecircuit scale. It is also possible to save electric power on processingthe moving image because the period for reducing the still image for thesake of configuring the moving image is limited to a VD period in whichall the pixels of the still image are being read out.

Furthermore, it is possible, when capturing the still image while takingthe moving image with an image sensing device requiring a mechanicalshutter, to reduce the frames blacking out as the moving image. Thus, itis possible to provide the image with little discomfort to the user asthe moving image.

Furthermore, it is possible to realize a process capable of handling anyhigh-pixel image sensing device. It is possible, by speeding up movingimage processing, to reduce the number of divisions of the still image,reduce the still image processing period, and increase the number offrames of large-screen still image continuous sensing. Further, it ispossible to cut down on the entire pixel memory amount if thelarge-screen still image continuous sensing is within a prescribednumber of frames.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is an overall configuration block diagram of an image sensingapparatus according to first, second, fourth and seventh embodiments ofthe present invention;

FIG. 2 is a configuration block diagram representing the first andseventh embodiments of the present invention;

FIG. 3 is a diagram showing a timing chart representing the firstembodiment of the present invention;

FIG. 4 is a configuration block diagram representing the secondembodiment of the present invention;

FIG. 5 is a diagram showing a timing chart representing the secondembodiment of the present invention;

FIG. 6 is an overall configuration block diagram of an image sensingapparatus representing a third embodiment of the present invention;

FIG. 7 is a configuration block diagram representing the thirdembodiment of the present invention;

FIG. 8 is a diagram showing a timing chart representing the thirdembodiment of the present invention;

FIG. 9 is a configuration block diagram representing the fourthembodiment of the present invention;

FIG. 10 is a diagram showing a timing chart representing the fourthembodiment of the present invention;

FIG. 11 is an overall configuration block diagram of an image sensingapparatus representing fifth and sixth embodiments of the presentinvention;

FIG. 12 is a configuration block diagram representing the fifth andsixth embodiments of the present invention;

FIG. 13 is a diagram showing a timing chart representing the fifthembodiment of the present invention;

FIG. 14 is a diagram showing a timing chart representing the sixthembodiment of the present invention;

FIG. 15 is a diagram showing a timing chart representing the seventhembodiment of the present invention.

FIG. 16 is a configuration block diagram showing a hardware structure ofa moving image and still image processing means of the presentinvention;

FIG. 17 is a diagram showing a table for a number representing thenumber of portions into which a still image is divided and a table for adelay time until the 1 starting of moving image processing of thepresent invention; and

FIG. 18 is a flowchart showing a sequence of steps at the moving imageand still image processing means of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment

FIGS. 1 to 3 are diagrams representing the first embodiment of thepresent invention. FIG. 1 is a block diagram representing the entireoverview of an image sensing apparatus of the present invention. FIG. 2is a block diagram showing characteristics relating to the firstembodiment of the present invention. FIG. 3 is a timing chart of variousprocesses representing the characteristics of this embodiment.

In the configuration order of FIG. 1, reference numeral 1 indicates animage sensing device which is a photoelectric conversion element. Thoughnot particularly limited as the image sensing device, it represents theimage sensing device such as a CCD or a CMOS sensor, which is the imagesensing device capable of reading out all the pixels within one frameperiod configuring a moving image as the first embodiment. Referencenumeral 2 indicates a lens. Reference numeral 3 indicates an ADconversion portion in control of AD conversion for digital-convertinganalog imaging data from the CCD or CMOS sensor, which is the imagesensing device 1, corresponding to a predetermined quantization bit.

Reference numeral 4 indicates a timing generator for controlling thetiming of the image sensing device 1 and the AD conversion portion 3.Reference numeral 5 indicates moving image and still image processingmeans for dividing the data AD converted by the AD conversion portion 3into moving image signals and still image signals to reduce and bufferthe images.

Reference numeral 6 indicates a memory for storing frame image data forstill image divided by the moving image and still image processing means5. In this embodiment, memory 6 can be RAM, which is just an example,and it is not limited thereto if it is an element having an access speedof a sufficient level.

Reference numeral 7 indicates an image processing portion for performinga processing procedure on moving image data and still image dataobtained from the moving image and still image processing means 5.Reference numeral 8 indicates a memory for temporarily storing processeddata and unprocessed data when processing the images in the imageprocessing portion 7. In this embodiment, memory 8 can be RAM which isjust an example, and it is not limited thereto if it is an elementhaving an access speed of a sufficient level.

Reference numeral 9 indicates a CPU as control means for controlling asystem of this embodiment. Reference numeral 10 indicates an encodeprocessing portion for compressing the data image-processed by the imageprocessing portion 7 according to a predetermined format. As the imageformat is compliant with a standard such as JPEG or MPEG, thisembodiment describes it as a JPEG encoder though it is not particularlylimited thereto.

Reference numeral 11 indicates switching means for switching whether torecord or communicate on the network the moving image data and stillimage data compressed by the encode processing portion 10. Referencenumeral 12 indicates a network communication portion for communicating acommunication command from an external network or delivering orcommunicating the moving image and still image processed in each of theblocks 1 to 11 to the outside.

Reference numeral 13 indicates a storage unit as storing means forstoring the moving image data or still image data in the image sensingapparatus of this embodiment. While FIG. 1 describes it as the storageunit, it indicates the one capable of data storage in general such as anonvolatile memory, a medium, or a hard disk capable of data writing,which may be replaceable and is not particularly limited.

Reference numeral 14 indicates an infrared remote control receivingportion for receiving a remote control command from outside. Referencenumeral 15 indicates a key input portion directly operable by the userfrom outside.

The configuration is configured on one chip when rendering as an LSI toinclude each of the blocks of the AD conversion portion 3, the timinggenerator 4, the moving image and still image processing means 5, theimage processing portion 7, the CPU 9, encode processing portion 10 andthe switching means 11. It is also possible, instead of rendering themas one chip, to configure each of the processing blocks separately orrender them as an LSI configuration which is convenient forimplementation of the devices.

Next, FIG. 2 is a block diagram for describing the blocks described asreference numerals 1 to 9 of FIG. 1 in detail, where reference numerals1, 3, 4, 6, 7 and 9 of FIG. 2 correspond to the aforementioned referencenumerals 1, 3, 4, 6, 7 and 9 of FIG. 1. A structure surrounded by a dotline in FIG. 2 corresponds to the moving image and still imageprocessing means 5 in FIG. 1.

Reference numeral 16 indicates storing means for storing readout datacorresponding to all the pixels of the image sensing device 1 with theimaging data AD converted by the AD conversion portion 3 as the stillimage data when capturing the still image in response to externalcontrol command input.

Reference numeral 17 indicates reduction processing means for reducingthe imaging data corresponding to all the pixels AD converted by the ADconversion portion 3 to an image size configuring the moving image.Reference numeral 18 indicates a buffer memory for holding the reducedimage data from the reduction processing means 17 for a predeterminedperiod and buffering the data.

Reference numeral 19 indicates selecting means for selecting imageprocessing of the moving image and the still image in response tocontrol from the CPU 9. Reference numeral 20 indicates a transmitter forgenerating a clock necessary for operation of this system, and referencenumeral 21 indicates a PLL circuit for multiplying a frequency of thetransmitter of the selecting means 19 to set it at a fast frequency.

A description will be provided that uses a timing chart of FIG. 3 todescribe the processing flow in the configuration of FIGS. 1 and 2.

The description of the signals and processes of FIG. 3 will start bydiscussing the signal and process shown at the top thereof and will thendiscuss the remaining signals and processes in descending vertical orderas shown in the figure. VD indicates a vertical synchronizing signal forconfiguring the moving image. Reference character b indicates the timingof reduction processing for creating an image configuring the movingimage and the buffering operation of the reduced data in the reductionprocessing means 17 and the buffer memory 8. Reference character aindicates the timing for storing and storing the data in the storingmeans for storing the still image by the entire pixel memory 6 andstoring means 16. Reference character c indicates the timing forperforming image processing in the image processing portion 7.

In the data flow in sensing the moving image normally, the period up tothe point described by a downward arrow as still image capture in FIG. 3represents a moving image sensing period. The timing between a VD signaland a next VD signal represents a unit frame period of the moving image.

(Moving Image Sensing Period)

The control means 9, via the timing generator 4, controls the imagesensing device 1 ready for reading out all the pixels to start readingout the imaging data corresponding to all the pixels. Next, according totiming of the timing generator 4, the data digitized by the ADconversion portion 3 undergoes image reduction by reduction processingmeans 17 and temporary buffering of the reduced image data by the buffer18 in the order of the pieces of the data read out. The reduced andbuffered image data is configured for the moving image.

The image data for the moving image temporarily buffered in the buffer18 is selected by the selecting means 19. After the elapse of apredetermined time from the start of moving image buffering shown in theportion of FIG. 3 indicated by reference character b, the imageprocessing shown in the portion of FIG. 3 indicated by referencecharacter c is sequentially performed on the moving images by imageprocessing means 7. In this case, the moving image processing startingtime is delayed by the predetermined time, and the moving imageprocessing is set in a time period not to overtake the data buffered inthe buffer 18, but to perform the moving image processing at a highspeed to an extent to keep it within a moving image frame. It is wellknown to set the predetermined delay time and control underrun oroverflow of the buffer 18, in accordance with the transfer rate of imagedata supplied to the buffer 18 and the transfer rate of image dataoutputted from the buffer 18.

In this case, a high-speed clock is supplied by the PLL circuit 21, andhigh-speed processing is realized by the image processing portion 7. Thecontents to be image-processed in the description indicate a normalprocessing procedure, which performs a filtering process of the imageand a luminance correction process, a color difference correctionprocess, and the like thereof to create an optimal image. As details ofthe processing contents are techniques heretofore known, and aredifferent depending on the points of view of manufacturers and the likerelating to the images, a description thereof will be omitted.

As the image processing portion 7 performs various kinds of imageprocessing operations, the memory 8 of FIG. 1 temporarily buffers theimage data that undergoes different kinds of image processing operationssequentially in chronological order by the image processing portion 7,and passes and receives the data to and from the image processingportion 7. Memory access and the like of the image processing portion isa technique heretofore known, and so a description thereof will beomitted.

Next, the data having completed the image processing operations isconfigured as one image within a frame period which is 1 VD period, andthe data is passed to the encode processing portion 10, which is thenext processing block. The encode processing portion 10 compresses theimage-processed data in the JPEG format as a preset and a predeterminedimage format. Whether the encoded data is stored in the storage areainside the image sensing apparatus or is communicated to an externaldestination via the network communication portion 12 is determined bythe switching means 11.

The image sensing apparatus is in such a state, because the CPU 9operates in a moving image sensing mode in response to a controlinstruction from outside. The destination of the moving image data isset to correspond to a control command controlled from outside inadvance, and is in a state of either communicating the moving image datato external devices via the network communication portion 12 or storingthe moving image data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting capture of a still image of a size larger thanthe image size of the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion 15 during the aforementioned moving imagesensing. In the case of receiving the command at the point described asstill image capture shown in FIG. 3, the storing means 16 starts storingoutput signals from the AD conversion portion 3 in readout orderstarting from a first VD (vertical synchronizing signal) after receivingthe capture command and stores them sequentially in the memory 6 untilthe readout is completed.

At the same time, the reduction and buffering operations of the movingimage data performed by the above described reduction processing means17 and the buffer 18 and the storing process of the still imageperformed by the storing means 16 are executed in parallel within thesame period. The data stored in the memory 6 for the still image ispassed, to perform the image processing of the still image data, to theimage processing portion 7 at a time other than the moving imageprocessing time in a period after counting predetermined VD periods(that is, including a next count) as shown in blanks in the sectionlabeled by the letter c in FIG. 3.

When image-processing the still image not to influence the moving imageprocessing time in the same image processing portion, it is not possibleto perform data processing of the entire data of one still image of asize larger than the moving image. Therefore, the still image isprocessed by having each frame of the still image divided into thepreset image areas and processing image data in the divided area withina non-processing time, as shown in a blank in the portion of FIG. 3identified by reference letter c, not performing the moving imageprocessing.

According to this embodiment, the moving image normally has a VGA size(640×480) while the still image is represented in a still image size(size corresponding to all the pixels) of 1280×960 which is twice aslarge vertically and horizontally. In this case, the number of dividedareas is four. In another case of the still image size corresponding toa larger image size for instance, the number of divided areas isincreased and the still image is processed in a unit of the divided areaas described above. In the case of an image size twice as largevertically and horizontally as 1280×960 for instance, the still image isprocessed by dividing it into 16 pieces. However, this is just anexample, and the number of divisions is set up according to the stillimage size, the moving image size, and so on.

Thus, the number of divisions corresponds to the number of VD frames sothat the image processing of the still image is completed within a timeperiod equivalent to the elapsing of 4 VDs in the case of 4 dividedareas. The moving image data then undergoes a JPEG image compressionprocess sequentially in an image compression processing portion which isthe encode processing portion 10 so as to be configured as a Motion-JPEGimage.

After the compression process, the moving image is stored in the storageunit 13 for storing the sensed moving image described above or has itsdata passed to the network communication portion 12. As for the stillimage, the divided image data is sequentially stored in the memory 8once by the image processing portion 7, and is passed to the encodeprocessing portion 10 as the still image after completely storing theimage data of one frame image so as to undergo a JPEG compressionprocess.

When a block of the still image to be encoded in this case is 1 block,after receiving a moving image communication or a storage finishcommand, it is delivered as one still image by the network communicationportion 12 in the case of communication, or is recorded as one stillimage in the storage in the case of the storage. In the case where theencode processing portion 10 is capable of high-speed processing,however, it may be arranged to compress one still image in anon-compression processing time of a moving image after image-processingof one still image finishes, but not after receiving the moving imagecommunication or the storage finish command.

In the case of receiving a still image capture command from outsideagain during a still image division processing (including a successivecapture request) in the above description, still image captureoperations are successively performed to the extent of not causing thecaptured image data to overflow the amounts of the memories 6 and 8.

As a first effect of realizing this embodiment as described above, it ispossible, even when capturing the still image of a larger size than themoving image while taking the moving image, to deliver the moving imageon the network or record the moving image without lowering the framerate of the moving image at all.

As a second effect, successive capture of large still images whiletaking the moving image becomes possible within the amount of the entirepixel memory 6. That is, successive capture during continuous VDsbecomes possible so as to obtain the images successively captured at thesame speed as moving image frames.

As a third effect, it is not necessary to have multiple different imageprocessing circuit blocks, one for the moving image and another for thestill image, and so the control function can be realized with no need toincrease the circuit scale on designing the circuits as an LSI.

As a fourth effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

Second Embodiment

FIGS. 1, 4 and 5 are diagrams representing a second embodiment of thepresent invention. A block diagram showing an image sensing apparatus ofthe second embodiment is the same configuration as that described inFIG. 1 of the first embodiment. FIG. 4 is a block diagram showing thecharacteristics relating to the second embodiment of the presentinvention. FIG. 5 is a timing chart representing the characteristics ofthis embodiment.

The configuration of FIG. 4 is a block diagram corresponding toreference numerals 1 to 21 of FIG. 2 described in the first embodiment,which is different therefrom in that the configuration does not have thebuffer memory 18 described in FIG. 2. The structure surrounded by a dotline in FIG. 4 corresponds to the moving image and the still imageprocessing means 5. The blocks of FIGS. 1 and 4 have the same functionsand configurations as the blocks of FIGS. 1 and 2 described in the firstembodiment, and so a description thereof will be omitted.

Therefore, the timing chart of FIG. 5 will be described hereinafter.

In descending vertical order of the signals and processes shown in FIG.5, VD indicates a vertical synchronizing signal for configuring themoving image, reference character d indicates a reduction processingoperation period of the reduction processing means 17, referencecharacter a indicates a timing for storing the still image data in theentire pixel memory 6 by the storing means 16, and reference character eindicates a timing for performing image processing in the imageprocessing portion 7.

Next, the contents of the present invention will be described along withthe timing chart.

(Moving Image Sensing Period)

First, in the data flow upon sensing the moving image normally, theperiod up to the point described by a downward arrow as the still imagecaptured in FIG. 5 represents a moving image sensing period. The timingbetween a VD signal and a next VD signal represents a unit frame periodof the moving image. The control means 9, via the timing generator 4,controls the image sensing device 1 to be ready for reading out all thepixels to start reading out the imaging data corresponding to all thepixels.

Next, according to the timing of the timing generator 4, the image datadigitized by the AD conversion portion 3 undergoes image reduction bythe reduction processing means 17 in the order of the pieces of theimage data read out so as to configure the processed image for themoving image. It is different from the first embodiment in that thereduced image is not buffered in the buffer 18.

Here, the moving image data is selected by the selecting means 19, andthe moving image processing shown in the portion of FIG. 5 identified byreference letter e is sequentially performed by the image processingportion 7 in the order of the moving image reduction processing shown inthe portion of FIG. 5 identified by reference letter d. The imageprocessing contents indicate the normal processing procedure, whichperforms a filtering process of the image and a luminance correctionprocess, a color difference correction process, and the like thereof tocreate an optimal image. Since details of the processing contents aretechniques heretofore known, and are different depending on the pointsof view of manufacturers and the like relating to the images, therefore,a description thereof will be omitted.

As the image processing portion 7 performs various kinds of imageprocessing, the memory 8 temporarily buffers the image data thatundergoes various kinds of image processing sequentially inchronological order by the image processing portion 7, and passes andreceives the data to and from the image processing portion 7. Memoryaccess and the like of the image processing portion 7 are performed by atechnique heretofore known, and so a description thereof will beomitted.

Next, the data having completed the image processing operations isconfigured as one image within a frame period which is 1 VD period, andthe data of one frame image is passed to the encode processing portion10, which is the next processing block. The encode processing portion 10compresses the image-processed data in the JPEG format as a preset andpredetermined image format. Whether the encoded data is stored in thestorage medium inside the image sensing apparatus or communicated to anexternal destination via the network communication portion 12 isdetermined by the switching means 11.

The image sensing apparatus is in such a state, because the CPU 9operates in a moving image sensing mode in response to a controlinstruction from outside. The destination of the moving image data isset to correspond to a control command controlled from outside inadvance, and is in a state of either communicating the moving image datato external devices via the network communication portion 12 or storingthe moving image data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting a capture of a still image of a size larger thanthe image size of the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion 15 during the aforementioned moving imagesensing.

In the case of receiving the command at the point described as the stillimage capture shown in FIG. 5, the storing means 16 stores the outputsignals from the AD conversion portion 3 in readout order starting fromthe first VD (vertical synchronizing signal) after receiving the capturecommand, and stores them sequentially in the entire pixel memory 6 bythe storing means 16 until the readout is completed.

At the same time, the reduction processing is sequentially executed bythe reduction processing means 17 in a timing of the periods of thereduction processing means illustrated in the portion of FIG. 5identified by reference letter d. The data stored for the still image inthe entire pixel memory 6 by the storing means 16 is passed, to performthe image processing of the still image, to the image processing portion7 in the non-processing time other than the moving image processing timein a period as shown in blanks in the portion of FIG. 5 identified byreference letter e after counting predetermined VD periods.

When image-processing the still image, it is not possible to pass theentire data of one frame still image. Therefore, the still imageprocessing is performed to each image area obtained by dividing thestill image into the preset image areas within a time which is anon-processing time during which the moving image processing is notperformed as shown in FIG. 5. The time approximately corresponds toretrace line period in the moving image frame period. Next, the stillimage data image-processed in the above description undergoes a JPEGimage compression process sequentially in the image compressionprocessing portion, which is the encode processing portion 10.

According to this embodiment, the moving image data undergoes the JPEGimage compression process sequentially in encode processing portion 10so as to be configured as a Motion-JPEG image. The moving image isstored in the storage unit 13 of FIGS. 1-13 for storing the sensedmoving image described above or has its data passed to the networkcommunication portion 12.

As for the still image, the divided image data is sequentially stored inthe memory 8 once by the image processing portion 7. Then, the stillimage data is passed to the encode processing portion 10 so as toundergo the JPEG compression process after completely storing the dataof one still image. When a block of the still image encoded in this caseis 1 block, after receiving a moving image communication or storagefinish command, it is delivered as one still image by the networkcommunication portion 12 in the case of communication, or is recorded asone still image in the storage unit 13 in the case of storage. In thecase where the encode processing portion 10 is capable of high-speedprocessing, however, it may be arranged to compress one still image in anon-compression processing time of a moving image after image-processingof one still image finishes, but not after receiving the moving imagecommunication or a storage finish command.

In the case of receiving a still image capture command from outsideagain during a still image division processing (including a successivecapture request) in the above description, still image captureoperations are successively performed to the extent of not causing thecaptured image data to overflow the amounts of the memories 6 and 8.

In the second embodiment, the non-processing time other than the movingimage processing time is shorter than the time described in the firstembodiment. However, it is possible to have an entirely satisfactoryprocessing time under this method when the image sizes of the movingimage and still image are smaller.

As a first effect of realizing this embodiment as described above, it ispossible, even when capturing the still image of a larger size than themoving image while taking the moving image, to deliver the moving imageon the network or record the moving image without lowering the framerate of the moving image at all.

As a second effect, successive capture of large still images whiletaking the moving image becomes possible within the amount of the entirepixel memory 6. That is, successive capture during continuous VDsbecomes possible so as to obtain the images successively captured at thesame speed as moving image frames.

As a third effect, it is not necessary to have multiple different imageprocessing circuit blocks, one for the moving image and another for thestill image, and so the control function can be realized with no need toincrease the circuit scale on designing the circuits as an LSI.

As a fourth effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

As a fifth effect, it is possible to further improve the third andfourth effects because the buffer 18 is not necessary for a movingimage.

Third Embodiment

FIGS. 6, 7 and 8 are diagrams representing a third embodiment of thepresent invention. FIG. 6 is a block diagram showing an overview of thisembodiment. FIG. 7 is a block diagram showing the characteristicsrelating to this embodiment. FIG. 8 is a timing chart of variousprocesses representing the characteristics of this embodiment.

In FIG. 6, elements having the same functions as FIG. 1 are given thesame numbers as those in FIG. 1. In FIG. 6, reference numeral 1indicates an image sensing device which is a photoelectric conversionelement. Though not particularly limited as the image sensing device, itrepresents an image sensing device, such as a CCD or a CMOS sensor,which is an image sensing device capable of reading out all the pixelswithin one frame period of a moving image according to this embodiment.Reference numeral 2 indicates a lens.

Reference numeral 3 indicates an AD conversion portion in control of ADconversion for digital-converting analog imaging data from the imagesensing device 1 corresponding to a predetermined quantization bit.Reference numeral 4 indicates a timing generator for controlling thetiming of the image sensing device 1 and the AD conversion portion 3.Reference numeral 7 indicates an image processing portion for performinga processing procedure on the image data AD converted by the ADconversion portion 3.

Reference numeral 8 indicates a memory for temporarily storing processeddata and unprocessed data when processing the images in the imageprocessing portion 7. This embodiment indicates the memory 8 to be anRAM, which is just an example, and it is not limited thereto if it is anelement having an access speed of a sufficient level. Reference numeral9 indicates a CPU as control means for controlling the system of thisembodiment.

Reference numeral 10 indicates an encode processing portion forcompressing the data processed by moving image and still imageprocessing means 24 according to a predetermined format. As the imageformat is compliant with a standard such as JPEG or MPEG, thisembodiment describes it as a JPEG encoder though it is not particularlylimited thereto.

Reference numeral 11 indicates switching means for determining whetherto record or communicate on the network the moving image data and stillimage data compressed by the encode processing portion 10. Referencenumeral 12 indicates a network communication portion for communicating acommunication command from an external network or delivering orcommunicating the moving image and still image processed in each of theprocessing portions to the outside.

Reference numeral 13 indicates a storage unit as storing means forstoring the moving image data or still image data in the image sensingapparatus of this embodiment. While FIG. 1 describes it as the storageunit, here reference numeral 13 indicates a storage unit capable of datastorage in general such as a nonvolatile memory, a medium or a hard diskcapable of data writing, which may be replaceable and is notparticularly limited. Reference numeral 14 indicates an infrared remotecontrol receiving portion for receiving a remote control command fromoutside.

Reference numeral 15 indicates a key input portion directly operable bythe user from outside. Reference numeral 23 indicates a memory which isimage data storing means for storing still image frame image datadivided by the moving image and still image processing means 24described later. This memory 23 in this embodiment indicates an RAM,which is just an example, and it is not limited thereto if it is anelement having an access speed of a sufficient level.

Reference numeral 24 indicates moving image and still image processingmeans for separating the image data processed by the image processingportion 7 into moving image signals and still image signals to reduceand/or buffer the images. Reference numeral 25 indicates image composingmeans for composing the image data (mainly still image data) switched bythe switching means 11.

The configuration is configured by one chip when rendering as an LSI toinclude each of the blocks of the AD conversion portion 3, the timinggenerator 4, image processing portion 7, the CPU 9, the encodeprocessing portion 10, the switching means 11, the moving image andstill image processing means 24 and the image synthesizing means 25 ofFIGS. 6-25. It is also possible, instead of rendering them as one-chip,to configure each of the processing blocks separately or render them asan LSI configuration which is convenient for implementation of thedevices.

Next, FIG. 7 is a block diagram for describing this embodiment indetail.

In FIG. 7, reference numeral 7 indicates an image processing means forperforming a processing procedure of the image data AD converted by theAD conversion portion 3 as in FIG. 6. Reference numeral 23 indicates amemory, which is the image data storing means for storing the stillimage frame image data stored in the storing means 26 described later.In this embodiment, memory 23 can be an RAM, which is just an example,and it is not limited thereto if it is an element having an access speedof a sufficient level.

Reference numeral 26 indicates storing means for storing the imageprocessing data processed by the image processing means 7 correspondingto the entire pixel data during 1 VD frame period as the still imagedata, when capturing the still image in response to external controlcommand input. Reference numeral 27 indicates reduction processing meansfor reducing the image-processed data corresponding to the entire pixeldata of the image sensing device 1 to the image size of the movingimage.

Reference numeral 28 indicates switching means for switching the imagecompression process of the moving image and the still image in responseto control from the CPU 9, which is the control means. Reference numeral29 indicates a transmitter for generating a clock necessary foroperation of this system. Reference numeral 30 indicates a PLL circuitfor multiplying the frequency of the transmitter 29 to set it at a fastfrequency.

A description will be provided by using the timing chart of FIG. 8 as tothe processing flow in the configuration of FIGS. 6 and 7.

In descending vertical order of the signals and processes of FIG. 8, VDindicates a vertical synchronizing signal for configuring the movingimage, reference character f indicates a timing for beingimage-processed in the image processing portion 7, reference character aindicates a timing for storing and storing the data in the entire pixelmemory 23 and the storing means 26 for storing the still image, andreference character g indicates a timing for reducing the image dataprocessed in the image processing portion 7 by the reduction processingmeans 27.

(Moving Image Sensing Period)

First, in a data flow upon sensing the moving image normally, the periodup to the point described by a downward arrow as the still image capturein FIG. 8 represents a moving image sensing period. The timing between aVD signal and a next VD signal represents a unit frame period of themoving image. The control means, via timing generator 4, controls theimage sensing device 1 to be ready for reading out all the pixels tostart reading out the imaging data corresponding to all the pixels.

Next, to perform the moving image processing shown in the portion ofFIG. 8 identified by reference letter f, the data digitized by the ADconversion portion 3 of all the pixels is sequentially image-processedby the image processing portion 7 in the order of the pieces of the dataread out according to the timing of the timing generator 4. The contentsto be image-processed indicate the normal processing procedure, whichperforms a filtering process of the image and a luminance correctionprocess, a color difference correction process, and the like thereof tocreate an optimal image. Since details of these processing operationsare heretofore known, and are different depending on the points of viewof manufacturers and the like relating to the images, a descriptionthereof will be omitted.

As the image processing portion 7 performs various kinds of imageprocessing operations, the memory 8 temporarily buffers the image datathat undergoes various kinds of image processing operations sequentiallyin chronological order by the image processing portion 7, and passes andreceives the data to and from the image processing portion 7. As memoryaccess and the like of the image processing portion 7 is a techniqueheretofore known, a description thereof will be omitted.

Next, the image of the image data, processed correspondingly to all thepixels by the image processing portion 7 and having undergone an imagereduction process by the reduction processing means 27, is configuredfor the moving image. The switching means 28 selects the moving image.The encode processing portion 10 has the reduced moving image reduced bythe reduction processing means 27 in the timing shown in the portion ofFIG. 8 identified by the reference character g sequentially compressedin the JPEG format as a preset and a predetermined image format. Theswitching means 11 determines whether to store the compressed data inthe storage unit 13 inside the image sensing apparatus or to communicatethe compressed data to an external destination via the networkcommunication portion 12.

As the image sensing apparatus is in such a state, the CPU 9 operates inresponse to a control instruction from outside to be in the moving imagesensing mode. The destination of the moving image data is set tocorrespond to a control command controlled from outside in advance, andis in a state of either communicating the moving image to externaldevices via the network communication portion 12 or storing the movingimage data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting a capture of a still image of a size larger thanthe image size configuring the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion 15 during the aforementioned moving imagesensing.

In the case of receiving the command at the point described as the stillimage capture shown in FIG. 8, the storing means 26 stores them from afirst VD (vertical synchronizing signal) after receiving the capturecommand in the order of undergoing the image processing in the imageprocessing portion 7 and stores them sequentially in the entire pixelmemory 23 as the storing means until the image processing is completed.At the same time, the images reduced by the reduction processing means27 are sequentially compressed to JPEG images by the encode processingportion 10 in a timing of the compression means shown in the portion ofFIG. 8 identified by reference letter g.

The data stored for the still image in the memory 23 is passed to theencode processing portion 10 to perform the image compression processingin the non-compression processing periods other than the moving imagecompression processing period as shown in blanks in the portion of FIG.8 identified by reference letter g after counting a predetermined VDperiod. When image-processing the still image, the compressionprocessing of the still image is performed on each portion of the imagedivided to correspond to the preset image areas by dividing the imagedata of one image into periods approximately corresponding to theretrace line periods of the moving image frame unit period which is anon-processing period during which the moving image processing operationis not performed as shown in FIG. 8.

According to this embodiment, the moving image data undergoes the JPEGimage compression process sequentially in the image compressionprocessing portion, which is the encode processing portion 10, so as tobe configured as a Motion-JPEG image. The moving image is stored in thestorage unit 13 for storing the sensed moving image described above orhas its data passed to the network communication portion 12.

As for the still image, the compressed and divided image data iscompletely synthesized as the image data of one image by the imagesynthesizing means 25. Thereafter, it is delivered as one still image bythe network communication portion 12 in the case of communication, or isrecorded as one still image in the storage unit 13 in the case of thestorage.

Here, the image data of the still image compressed by the encodeprocessing portion 10 is synthesized as one image so as to be deliveredto the network or recorded in the storage unit. It is also possible,however, to deliver the compressed and divided image in an as-is dividedstate to the network or record it in the storage unit.

In the case of receiving a still image capture command from outsideagain during the still image division processing period (including acontinuous sensing request) in the description of this embodiment, astill image capture operation is continuously performed in sequence tothe extent of not causing the amount of the entire pixel memory 23 tooverflow.

As the first effect of realizing this embodiment as described above aswith the first embodiment, it is possible, when capturing the stillimage of a larger size than the moving image while taking the movingimage, to deliver it on the network or record it without lowering theframe rate of the moving image at all.

As the second effect, continuous sensing of large-screen still imageswhile taking the moving image becomes possible within the amount of theentire pixel memory 6 (continuous sensing by VD becomes possible so asto obtain the images continuously shot at the same speed as the movingimage frames).

As the third effect, it is not necessary to have multiple imageprocessing circuit blocks, and so the control can be realized with noneed to increase the circuit scale on rendering it as an LSI.

As the fourth effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

As the fifth effect, it is possible to further promote the third andfourth effects because a reduced image buffer is not necessary formoving image configuration.

Fourth Embodiment

FIGS. 1, 9 and 10 are diagrams representing a fourth embodiment of thepresent invention. A block diagram showing an image sensing apparatus ofthe second embodiment is the same configuration as that described inFIG. 1 of the first embodiment. FIG. 9 is a block diagram showing thecharacteristics relating to this embodiment. FIG. 10 is a timing chartof various processes representing the characteristics of thisembodiment. Here, the description of FIG. 1 will be omitted.

FIG. 9 is a block diagram for describing the blocks described asreference numerals 1 to 9 of FIG. 1 in detail, where reference numerals1, 3, 4, 6, 7 and 9 of FIG. 9 correspond to the aforementioned referencenumerals 1, 3, 4, 6, 7 and 9 of FIG. 1. The structure surrounded by adot line in FIG. 9 corresponds to the moving image and still imageprocessing means 5 of FIG. 1.

Reference numeral 17 of FIG. 9 indicates reduction processing means forreducing the image data corresponding to all the pixels read out by thestill image readout channel to the image size of the moving image.Reference numeral 15 indicates storing means for storing the entirepixel readout data corresponding to the still image mode of the imagesensing device 1.

Reference numeral 18 indicates a buffer memory for holding the reducedimage data from the reduction processing means 17 for a predeterminedperiod and buffering the data. Reference numeral 19 indicates selectingmeans for selecting image processing of the moving image and the stillimage in response to control from the CPU 9 as control means. Referencenumeral 20 indicates a transmitter for generating a clock necessary foroperation of this system. Reference numeral 21 indicates a PLL circuitfor multiplying a frequency of the transmitter 20 to set it at a fastfrequency.

Reference numeral 31 indicates switching means for switching a flow of adata processing block of the imaging data AD converted by the ADconversion portion 3 to correspond to readouts of the moving imagereadout channel and the still image readout channel. Reference numeral32 indicates a buffer for buffering the image read out in a moving imagereadout mode by the switching means 31.

As for the configurations of FIGS. 1 and 9, the flow of processing willbe described by referring to the timing chart shown in FIG. 10.

In descending vertical order of the signals and processes of FIG. 10, VDindicates a vertical synchronizing signal for configuring the movingimage, reference character h indicates a buffering operation timing ofthe moving image data read out of a normal moving image readout channelby the buffer 32, reference character i indicates a timing for storingand storing the data in the entire pixel memory 6 and storing means 16for storing the still image, reference character j indicates a bufferingoperation timing of the reduction process for creating an imageconfiguring the moving image with the reduction processing means 17 andbuffer 18 and the reduced data, and reference character k indicates atiming for performing the image processing with the image processingportion 7.

(Moving Image Sensing Period)

First, in a data flow upon sensing the moving image normally, the periodup to the point described by a downward arrow as the still image capturein FIG. 9 represents a moving image sensing period. The timing between aVD signal and a next VD signal represents a unit frame period of themoving image. The control means 9 controls the timing generator 4 tostart reading out pixel imaging data corresponding to the moving imagesize, in the moving image readout channel setup.

Next, the data is inputted to the switching means 31 in the order of thepieces of the data digitized by the AD conversion portion 3 read outaccording to the timing of the timing generator 4. The switching means31 is switched to the moving image mode on sensing the moving imagenormally.

The image data for the moving image temporarily buffered in the buffermemory 32 has its moving image selected by the selecting means 19. Afterthe elapse of a predetermined time from the start of moving imagebuffering shown in the portion of FIG. 10 identified by reference letterh, the moving image processing shown in the portion of FIG. 10identified by reference letter k is sequentially performed to the movingimages by the image processing means 7. In this case, as the movingimage processing starting time is delayed by the predetermined time, themoving image processing is set in a time period not to overtake the databuffered in the buffer 32 so as to perform the moving image processingat a high speed to the extent of keeping it within a moving image frame.In this case, a high-speed clock is supplied by the PLL circuit 21 tohave high-speed processing realized by the image processing portion 7.

The contents to be image-processed in the description indicate a normalprocessing procedure, which performs a filtering process of the imageand a luminance correction process, a color difference correctionprocess, and the like thereof to create an optimal image. As details ofthe processing contents are techniques heretofore known, and aredifferent depending on the points of view of manufacturers and the likerelating to the images and so a description thereof will be omitted. Asthe image processing portion 7 performs various kinds of imageprocessing operations, the memory 8 temporarily buffers the image datathat undergoes various the kinds of image processing operationssequentially in chronological order by the image processing portion 7,and passes and receives the data to and from the image processingportion 7. As memory access and the like of the image processing portionis a technique heretofore known, a description thereof will be omitted.

Next, the data having completed the image processing operations isconfigured as one image within a frame period which is 1 VD period, andthe data is passed to the encode processing portion 10, which is thenext processing block. The encode processing portion 10 compresses theimage-processed data in the JPEG format as a preset and predeterminedimage format, and the data is stored in the storage unit inside theimage sensing apparatus by the switching means 11 or communicated to anexternal destination via the network communication portion 12.

As the image sensing apparatus is in such a state, the CPU 9 operates inresponse to a control instruction from outside to be in the moving imagesensing mode. The destination of the moving image data is setcorrespondingly to a control command controlled from outside in advance,and is in a state of either communicating the moving image to theexternal devices via the network communication portion 12 or storing themoving image data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting a capture of a still image of a size larger thanthe image size configuring the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion during the aforementioned moving imagesensing.

In the case of receiving the command at the point described as the stillimage capture shown in FIG. 10, the image sensing device 1 is switchedto readout of the entire pixel readout channel for dealing with thestill image readout starting from the first VD (vertical synchronizingsignal) after receiving the capture command. The switching isimplemented by switching the readout timing of the timing generator 4.At the same time, the data is passed to the storing means 16 and thereduction processing means 17 for the sake of the still image readout bythe switching means 31. This frame is set as a still image data storageframe, which starts storing the output signals from the AD conversionportion 3 in the storing means 16 in readout order, and stores themsequentially in the memory 6 as the storing means until the readout iscompleted.

At the same time, the reduction and buffering of the image of the samesize as the moving image channel read out in a previous VD period by theabove described reduction processing means 17 and the buffer 18 areprocessed in parallel. An image having the still image reduced is usedfor the moving image data in 1 VD frame period following receipt of acapture control signal.

Thereafter, it is switched from an immediate VD frame to the abovedescribed moving image mode readout channel to perform the moving imageprocessing sequentially in the image processing portion 7. Still imagestorage data stored as the still image data in the memory 6 is passed tothe image processing portion 7 to perform the image processing inperiods other than a moving image processing period after countingpredetermined VD periods (that is, including a next count) as shown inblanks in the portion of FIG. 9 identified by reference letter k.

When image-processing the still image not to influence the moving imageprocessing time in the same image processing portion, it is not possibleto perform data processing of the entire data of one still image of asize larger than the moving image. Therefore, the still image isprocessed by having each portion of the image divided to correspond tothe preset image areas divided over non-processing periods notperforming the moving image processing as shown in FIG. 3.

According to this embodiment, the moving image read out by the movingimage read out channel has a VGA size (640×480) while the still imageread out by the still image read out channel is represented in a stillimage size (size corresponding to all the pixels) of 1280×960 which istwice as large vertically and horizontally. In the case, a number ofdivision is four. In another case of the still image size correspondingto a larger image size for instance, the number of divided areas isincreased and the still image is processed as described above. In thecase of the image size being twice as large vertically and horizontallyas 1280×960, for instance, the still image is processed by dividing itinto 16 pieces. However, this is just an example, and the number ofdivisions is set up according to the still image size and the movingimage size.

The moving image data then undergoes a JPEG image compression processsequentially in the image compression processing portion which is theencode processing portion 10 so as to be configured as a Motion-JPEGimage. After the compression process, the moving image is stored in thestorage unit 13 for storing the moving image sensing described above orhas its data passed to the network communication portion 12.

As for the still image, the divided image data is sequentially stored inthe memory 8 once by the image processing portion 7, and is passed tothe encode processing portion 10 as the still image after completelystoring the image data of one image so as to undergo the JPEGcompression process. When an encode processing block of the still imagein this case is 1 block, after receiving a moving image communication orstorage finish command, it is delivered as one still image by thenetwork communication portion 12 in the case of communication, or isrecorded as one still image in the storage in the case of the storage.In the case where the encode processing portion 10 is a block capable ofhigh-speed processing, however, it may be arranged to compress one imagein a non-compression processing period of a moving image compressionprocess after image-processing one image without performing the storageor network delivery after receiving the storage command.

In the case of receiving a still image capture command from outsideagain during a still image division processing period (including acontinuous sensing request) in the above description, a still imagecapture operation is continuously performed to the extent of not causingthe amounts of the memories 6 and 8 to overflow.

As the first effect of realizing this embodiment as described above aswith the first embodiment, it is possible, when capturing the stillimage of a larger size than the moving image while taking the movingimage, to deliver it on the network or record it without lowering theframe rate of the moving image at all.

As the second effect, continuous sensing of large-screen still imageswhile taking the moving image becomes possible within the amount of theentire pixel memory 6. That is, continuous sensing by VD becomespossible so as to obtain the images continuously shot at the same speedas moving image frames.

As the third effect, it is not necessary to have multiple imageprocessing circuit blocks, and so the control can be realized with noneed to increase the circuit scale on rendering it as an LSI.

As the fourth effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

As the fifth effect, it is possible to save electric power on processingthe moving image because a period for reducing the still image for thesake of configuring the moving image is limited to a VD period in whichall the pixels of the still image are being read out.

Fifth Embodiment

FIGS. 11, 12 and 13 are diagrams representing a fifth embodiment of thepresent invention. FIG. 11 is a block diagram representing an overviewof the image sensing apparatus of the present invention. FIG. 12 is ablock diagram showing the characteristics relating to this embodiment.FIG. 13 is a timing chart of various processes representing thecharacteristics of this embodiment.

In configuration order of FIG. 11, reference numeral 1 indicates animage sensing device, which is a photoelectric conversion element.Though not particularly limited as the image sensing device, itrepresents an image sensing device, such as a CCD or a CMOS sensor. Theimage sensing device has a moving image readout channel for dealing withthe moving image and a still image readout channel for dealing with thestill image readout according to this embodiment. The moving imagereadout channel is a channel capable of reading the data havingundergone pixel skipping or having the pixels added correspondingly tothe moving image size out of the entire pixel data. The still imagereadout channel is a channel requiring an exposure period for the stillimage and is capable of reading the data of all the pixels in multipleframe periods of the moving image, which are periods different from theexposure period. It is an image sensing device capable of reading allthe pixels in 2 frame periods with a sensor of 3 to 4 million pixels orso.

The configuration of the elements denoted by reference numerals 2 to 15of FIG. 11 corresponds to the configuration of the elements denoted byreference numerals 2 to 15 of FIG. 1 according to the first embodiment.Reference numeral 33 indicates a mechanical shutter for physicallyshutting out external light during a sensor readout period of the imagesensing device 1.

The configuration is configured by one chip when rendering as an LSI toinclude each of the blocks of the AD conversion portion 3, the timinggenerator 4, the moving image and still image processing means 5, theimage processing portion 7, the CPU 9, the encode processing portion 10and the switching means 11. It is also possible, instead of renderingthem as one-chip, to configure each of the processing blocks separatelyor render them as an LSI configuration, which is convenient forimplementation of the devices.

Next, FIG. 12 is a block diagram for describing the blocks described asreference numerals 1 to 9 of FIG. 11 in detail, where reference numerals1, 3, 4, 6, 7 and 9 of FIG. 12 correspond to the aforementionedreference numerals 1, 3, 4, 6, 7 and 9 of FIG. 1. The structuresurrounded by a dot line in FIG. 12 corresponds to the moving image andstill image processing means 5 of FIG. 11.

Reference numeral 17 indicates reduction processing means for reducingthe image data corresponding to all the pixels read out by the stillimage readout channel to the image size of the moving image. Referencenumeral 15 indicates storing means for storing the entire pixel readoutdata corresponding to the still image mode of the image sensing device1.

Reference numeral 18 indicates a buffer memory for holding the reducedimage data from the reduction processing means 17 for a predeterminedperiod and buffering the data. Reference numeral 19 indicates selectingmeans for selecting image processing of the moving image and the stillimage in response to control from the CPU 9 as control means. Referencenumeral 20 indicates a transmitter for generating a clock necessary foroperation of this system, and reference numeral 21 indicates a PLLcircuit for multiplying a frequency of the transmitter 20 to set it at afast frequency.

Reference numeral 31 indicates switching means for switching a flow of adata processing block of the imaging data AD converted by the ADconversion portion 3 to correspond to readouts of the moving imagereadout channel and the still image readout channel. Reference numeral32 indicates a buffer memory for buffering the image read out in amoving image channel readout mode by the switching means 31.

Reference numeral 33 corresponds to reference numeral 33 of FIG. 11, andindicates a mechanical shutter for physically shutting out externallight during a sensor readout period of the image sensing device 1. Adescription will be provided by using the timing chart shown in FIG. 13as to the processing flow in the configuration of FIGS. 11 and 12.

In descending vertical order of the signals and processes of FIG. 13, VDindicates a vertical synchronizing signal for configuring the movingimage, reference character l indicates a timing for opening and closingthe mechanical shutter 33, reference character m indicates a timingrepresenting a readout mode of the CCD of the image sensing device 1,reference character n indicates a timing for storing and storing thedata in the entire pixel memory 6 and storing means 16 for storing thestill image, reference character o indicates a buffering operationtiming of the moving image data read out of a normal moving imagereadout channel by the buffer memory 32, reference character p indicatesa buffering operation timing of the reduction process for creating animage configuring the moving image from the still image data read out ofthe still image readout channel of the CCD with the reduction processingmeans 17 and the buffer 18 and the reduced data, and reference characterq indicates a timing for performing the image processing with the imageprocessing portion 7.

(Moving Image Sensing Period)

First, in a data flow on sensing the moving image normally, the periodup to the arrow portion below the point described as the still imagecapture in FIG. 13 represents a moving image sensing period, and thetiming between a VD signal and a next VD signal represents a unit frameperiod configuring the moving image. The timing generator 4 of FIGS.12-4 is controlled in the moving image readout channel setup with themechanical shutter 33 open so as to start reading out pixel imaging datacorresponding to the moving image size.

Next, the data is inputted to the switching means 31 in the order of thepieces of the data digitized by the AD conversion portion 3 read outaccording to the timing of the timing generator 4. The switching means31 is switched to the moving image mode on sensing the moving imagenormally.

The image data for the moving image temporarily buffered in the buffermemory 32 has its moving image selected by the selecting means 19. Afterthe elapse of a predetermined time from the start of moving imagebuffering in the portion of FIG. 13 identified by reference letter o,the moving image processing shown in the portion of FIG. 13 identifiedby reference letter q is sequentially performed by image processingmeans 7. In this case, as the moving image processing starting time isdelayed by the predetermined time, the moving image processing is set ina time period not to overtake the data buffered in the buffer 32 so asto perform the moving image processing at a high speed to the extent ofkeeping it within a moving image frame. In this case, a high-speed clockis supplied by the PLL circuit 21 to have high-speed processing realizedby the image processing portion 7.

The contents to be image-processed in the description indicate a normalprocessing procedure, which performs a filtering process of the imageand a luminance correction process, a color difference correctionprocess, and the like thereof to create an optimal image. As details ofthe processing operations are techniques heretofore known, and aredifferent depending on the points of view of manufacturers and the likerelating to the images, a description thereof will be omitted.

As the image processing portion 7 performs various kinds of imageprocessing, the memory 8 temporarily buffers the image data undergoingvarious kinds of image processing operations sequentially inchronological order by the image processing portion 7, and passes andreceives the data to and from the image processing portion 7. As memoryaccess and the like of the image processing portion is a techniqueheretofore known, a description thereof will be omitted.

Next, the data having completed the image processing operations isconfigured as one image within a frame period which is 1 VD period, andthe data is passed to the encode processing portion 10, which is thenext processing block. The encode processing portion 10 compresses theimage-processed data in the JPEG format as a preset and predeterminedimage format, and the data is stored in the storage unit 13 inside theimage sensing apparatus by the switching means 11 or communicated to anexternal destination via the network communication portion 12.

As the image sensing apparatus is in such a state, the CPU 9 operates inresponse to a control instruction from outside to be in the moving imagesensing mode. The destination of the moving image data is set tocorrespond to a control command controlled from outside in advance, andis in a state of either communicating the moving image to the externaldevices via the network communication portion 12 or storing the movingimage data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting a capture of a still image of a size larger thanthe image size of the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion 15 during the aforementioned moving imagesensing operation.

In the case of receiving the command at the point described as the stillimage capture shown in FIG. 13, the CCD as the image sensing device 1 isset to an entire pixel exposure period for dealing with the still imagereadout over the first VD (vertical synchronizing signal) period afterreceiving the capture command. The mechanical shutter 33 remains openduring that period as shown in the portion of FIG. 13 identified byreference letter l, and no data is passed to the other processingcircuit blocks during the exposure period of the CCD.

The CCD as the image sensing device 1 can read out the image data inresponse to the next VD frame signal. Here, the mechanical shutter 33opened during the exposure period is closed in synchronization with theVD signal. It starts reading out the data exposed for 1 VD period from atime point when a ½ image of the image data corresponding to the stillimage is synchronized with the VD signal of the part of (1) described inFIG. 13, and has the data sequentially stored by the storing means 15 inthe memory 6. The pixel data of the remaining ½ portion is read out inthe VD period of the part of (2) described in FIG. 13, and is stored bythe storing means 15 in the memory 6 as with the above described ½pixels.

In the portion of FIG. 13 identified by reference letter p, the image isreduced by the reduction processing means 17 and the reduced image istemporarily buffered by the buffer 18 in parallel with the still imageprocessing operation as from the time point indicated by (1)+(2) so asto configure the processed image for the moving image.

The image data for the moving image temporarily buffered by the buffer18 has its moving image selected by the selecting means 19. After theelapse of a predetermined time from the start of moving image bufferingin the portion of FIG. 13 identified by reference letter b, the movingimage processing shown in the portion of FIG. 13 identified by referenceletter q is sequentially performed by image processing means 7. In thiscase, as the moving image processing starting time is delayed by thepredetermined time, the moving image processing is set in a time periodnot to overtake the data buffered in the buffer 18 so as to perform themoving image processing at a high speed to an extent of keeping itwithin a moving image frame.

Thereafter, the mechanical shutter 33 is opened in synchronization withimmediate VD signal timing. At the same time, a switch is made from theVD frame to the above described moving image mode readout channel so asto perform the moving image processing sequentially in the imageprocessing portion 7.

Still image storage data stored as the still image data in the memory 6is passed to the image processing portion 7 to perform the imageprocessing in periods other than the moving image processing periodafter counting a predetermined VD period (that is, including a nextcount) as shown in blanks in the portion of FIG. 13 identified byreference letter q.

When image-processing the still image not to influence the moving imageprocessing time in the same image processing portion, it is not possibleto perform data processing of the entire data of one still image of asize larger than the moving image. Therefore, the still image isprocessed by having each portion of the image divided to correspond tothe preset image areas divided over non-processing periods notperforming the moving image processing as shown in FIG. 13.

Here, a description will be provided as to the mode for reading out allthe pixels corresponding to the still image by exemplifying the case ofthe CCD of a primary-color filter and RGB pixel arrays of a Bayer arrayinterlace readout method. When reading out the still image, it isarranged to sequentially read out R, Gr, R, Gr . . . which are signalcomponents of pixel odd-numbered lines in a first field and Gb, B, Gb, B. . . which are signal components of pixel even-numbered lines in asecond field.

As the image of only the first field is read in 1 VD period of themoving image frame period and the remaining image of the second field isread in the following VD period, only the R component and Gr componentcan be read in the 1 VD period. Therefore, it is arranged to move on tothe next process after referring to the data after sequentially readingout Gb, B, Gb, B . . . from the start of readout of the second field andthe data stored in the memory 6.

The processing flow and processing timing of each of the blocks 10 to 13of FIG. 11 from the image processing portion 7 onward correspond to theprocessing contents of the blocks 10 to 13 of FIG. 1 according to thefirst embodiment so as to perform exactly the same process.

According to this embodiment, the frame configuring the moving imageduring a CCD exposure period and the first frame period on reading outthe CCD are in a state not capable of updating the image data so thatthe moving image blacks out only in these periods. In the moving imageblackout period, the images in the period immediately before the CCDexposure frame period are sequentially rendered as the moving images.

As the first effect of realizing this embodiment as described above, itis possible, when capturing the still image while taking the movingimage with the image sensing device 1 requiring the mechanical shutter33, to reduce the frames blacking out as the moving image so as toprovide the image with little discomfort to the user as the movingimage.

As the second effect, it is not necessary to have multiple imageprocessing circuit blocks, and so the control can be realized with noneed to increase the circuit scale on rendering it as an LSI.

As the third effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

Sixth Embodiment

FIGS. 11, 12 and 14 are diagrams representing a sixth embodiment of thepresent invention. A block diagram representing an overview of the imagesensing apparatus of the present invention is the same as the fifthembodiment in FIG. 11, and a block diagram showing the characteristicsrelating to this embodiment is the same as the fifth embodiment in FIG.12. FIG. 14 is a timing chart of various processes representing thecharacteristics of this embodiment.

A description will be provided by using a timing chart shown in FIG. 14as to the processing flow in the configuration of FIGS. 11 and 12.

In descending vertical order of the signals and processes of FIG. 14, VDindicates a vertical synchronizing signal for configuring the movingimage, reference character l indicates a timing for opening and closingthe mechanical shutter 33, reference character m indicates a timingrepresenting a readout mode of the CCD as the image sensing device 1,reference character n indicates a timing for storing and storing thedata in the entire pixel memory 6 and storing means 16 for storing thestill image, reference character o indicates a buffering operationtiming of the moving image data read out of a normal moving imagereadout channel by the buffer memory 32, and reference character pindicates a buffering operation timing of the reduction process forcreating an image configuring the moving image from the still image dataread out of the still image readout channel of the CCD with thereduction processing means 17 and the buffer 18 and the reduced data.Reference character q indicates a timing for performing the imageprocessing with the image processing portion 7.

(Moving Image Sensing Period)

First, in a data flow upon sensing the moving image normally, the periodup to the arrow portion below the point described as the still imagecapture in FIG. 14 represents a moving image sensing period, and thetiming between a VD signal and a next VD signal represents a unit frameperiod configuring the moving image. The timing generator 4 iscontrolled in the moving image readout channel setup with the mechanicalshutter 33 open so as to start reading out pixel imaging datacorresponding to the moving image size.

Next, the data is inputted to the switching means 31 in the order of thepieces of the data digitized by the AD conversion portion 3 read outaccording to the timing of the timing generator 4. The switching means31 is switched to the moving image mode upon sensing the moving imagenormally.

The image data for the moving image temporarily buffered in the buffermemory 32 has its moving image selected by the selecting means 19. Afterthe elapse of a predetermined time from the start of moving imagebuffering in the portion of FIG. 14 indicated by reference letter o, themoving image processing shown in the portion of FIG. 13 indicated byreference letter q, is sequentially performed by image processing means7. In this case, as the moving image processing starting time is delayedby the predetermined time, the moving image processing is set in a timeperiod not to overtake the data buffered in the buffer 32 so as toperform the moving image processing at a high speed to an extent ofkeeping it within a moving image frame.

In this case, a high-speed clock is supplied by the PLL circuit 21 tohave high-speed processing realized by the image processing portion 7.The contents to be image-processed in the description indicate a normalprocessing procedure, which performs a filtering process of the imageand a luminance correction process, a color difference correctionprocess, and the like thereof to create an optimal image. As details ofthe processing contents are techniques heretofore known, and aredifferent depending on points of view of manufacturers and the likerelating to the images, a description thereof will be omitted.

As the image processing portion 7 performs various kinds of imageprocessing, the memory 8 temporarily buffers the image data thatundergoes various kinds of image processing operations sequentially inchronological order by the image processing apparatus 7, and passes andreceives the data to and from the image processing portion 7. As memoryaccess and the like of the image processing portion is a techniqueheretofore known, a description thereof will be omitted.

Next, the data having completed the image processing operations isconfigured as one image within a frame period that is 1 VD period, andthe data is passed to the encode processing portion 10, which is thenext processing block. The encode processing portion 10 compresses theimage-processed data in the JPEG format as a preset and predeterminedimage format, and the data is stored in the storage unit inside theimage sensing apparatus by the switching means 11 or communicated to anexternal destination via the network communication portion 12.

As the image sensing apparatus is in such a state, the CPU 9 operates inresponse to a control instruction from outside to be in the moving imagesensing mode. The destination of the moving image data is setcorrespondingly to a control command controlled from outside in advance,and is in a state of either communicating the moving image to theexternal devices via the network communication portion 12 or storing themoving image data sequentially in the storage unit 13.

(Capture Still Image)

A description will be provided as to the case where the CPU 9 receives acommand for requesting a capture of a still image of a size larger thanthe image size configuring the moving image from any one of the networkcommunication portion 12, the infrared remote control receiving portion14 and the key input portion 15 during the aforementioned moving imagesensing.

In the case of receiving the command at the point described as the stillimage capture shown in FIG. 14, the CCD as the image sensing device 1 isset to an entire pixel exposure period for dealing with the still imagereadout over the first 1 VD (vertical synchronizing signal) period afterreceiving the capture command. The mechanical shutter 33 remains openduring that period as shown in the portion of FIG. 14 indicated byreference letter l, and no data is passed to the other processingcircuit blocks during the exposure period of the CCD.

The CCD as the image sensing device 1 can read out the image data inresponse to the next VD frame signal. Here, the mechanical shutter 33opened during the exposure period is closed in synchronization with theVD signal. It starts reading out the data exposed for 1 VD period from atime point when a ⅓ image of the image data corresponding to the stillimage is synchronized with the VD signal of the part of (1) described inFIG. 14, and has the data sequentially stored by the storing means 15 inthe memory 6.

The pixel data of the ⅓ portion is read out in the VD period of the partof (2) described in the following FIG. 14, and is sequentially stored bythe storing means 15 in the memory 6 as with the above described ⅓pixels of the part of (1). The pixel data of the last ⅓ portion is readout in the VD period of the part of (3) described in the following FIG.14, and is sequentially stored by the storing means 15 in the memory 6as with the above described ⅓ pixels of the part of (3).

In the portion of FIG. 14 indicated by reference letter p, the image isreduced by the reduction processing means 17 and the reduced image istemporarily buffered by the buffer 18 in parallel with the still imageprocessing operation as from the time point indicated by (1)+(2)+(3) soas to configure the processed image for the moving image. The image datafor the moving image temporarily buffered by the buffer 18 has itsmoving image selected by the selecting means 19. After elapse of apredetermined time from the start of moving image buffering in theportion of FIG. 3 indicated by reference letter b, the moving imageprocessing shown in the portion of FIG. 13 indicated by reference letterq is sequentially performed by image processing means 7. In this case,as the moving image processing starting time is delayed by thepredetermined time, the moving image processing is set in a time periodnot to overtake the data buffered in the buffer 18 so as to perform themoving image processing at a high speed to an extent of keeping itwithin a moving image frame.

Thereafter, the mechanical shutter 33 is opened in synchronization withimmediate VD signal timing. At the same time, a switch is made from theVD frame to the above described moving image mode readout channel so asto perform the moving image processing sequentially in the imageprocessing portion 7.

Still image storage data stored as the still image data in the memory 6is passed to the image processing portion 7 to perform the imageprocessing in periods other than the moving image processing periodafter counting a predetermined VD period (that is, including a nextcount) as shown in blanks in the portion of FIG. 14 indicated byreference letter q.

When image-processing the still image not to influence the moving imageprocessing time in the same image processing portion, it is not possibleto perform data processing of the entire data of one still image of asize larger than the moving image. Therefore, the still image isprocessed by having each portion of the image divided correspondingly tothe preset image areas divided over non-processing periods notperforming the moving image processing as shown in FIG. 14.

Here, a description will be provided as to the mode for reading out allthe pixels corresponding to the still image by exemplifying the case ofthe CCD of a primary-color filter and RGB pixel arrays of a Bayer arrayinterlace readout method. It is a method of, when reading out the stillimage, reading out the pixels across three fields, where a first fieldline, a second field line and a third field line are configured bysignal components. A first line of the first field is read out in orderof R, Gr, R, Gr . . . , and a second line of the first field is normallya readout of a fourth line of CCD pixels (1 (first line)+3 (fields))which is in order of Gb, B, Gb, B.

A next line is inversely arranged to have different reference data ineach of the lines of the same field, such as R, Gr, R, Gr. Similarly,the first line of the second field is configured in reverse readoutorder to the first field. Similarly, the third field is arranged to beread out in the same order as the first field, that is, in order of athird line of the CCD pixels to the signal components of the pixelsdisplaced by +3 lines.

The image of only the first field is read in 1 VD period of the movingimage frame period, only the image of the second field is read in thefollowing VD period, and the image of the third field is read in thefollowing VD period. Thus, the pixels of the entire screen cannot beread out in 2 VD periods from the start of the readout. Therefore, it isarranged to move on to the next process after referring to the dataafter sequentially reading it out from the start of readout of the thirdfield and the data stored in the memory 6.

The processing flow and processing timing of each of the blocks 10 to 13of FIG. 11 from the image processing portion 7 onward correspond to theprocessing contents of the blocks 10 to 13 of FIG. 1 according to thefirst embodiment so as to perform exactly the same process.

According to this embodiment as with the fifth embodiment, a total ofthree frame periods of an exposure frame configuring the moving imageduring a CCD exposure period and the first frame and second frameperiods configuring the moving image on reading out the CCD are in astate not capable of updating the image data so that the moving imageblacks out only in these periods. In the moving image blackout period,the images in the period immediately before the CCD exposure frameperiod are sequentially rendered as the moving images.

As the first effect of realizing this embodiment as described above, itis possible, when capturing the still image while taking the movingimage with the image sensing device 1 requiring the mechanical shutter33, to reduce the frames blacking out as the moving image so as toprovide the image with little discomfort to the user as the movingimage.

As the second effect, it is not necessary to have multiple imageprocessing circuit blocks, and so the control can be realized with noneed to increase the circuit scale on rendering it as an LSI.

As the third effect, it is possible to reduce the power consumption ofthe circuits by preventing an increase in the circuit scale.

As the fourth effect, it is possible to realize a process capable ofdealing with any high-pixel image sensing device.

Seventh Embodiment

FIGS. 1, 2 and 15 are diagrams representing a seventh embodiment of thepresent invention. FIG. 1 has the same configuration as that accordingto the first embodiment, and FIG. 2 also has the same configuration asthat according to the first embodiment. The difference lies in thecontents is the PLL circuit 21 of FIG. 2, which are arranged to have ahigher multiplication rate than the PLL circuit 21 shown in the firstembodiment and speed up the operation of the blocks operated by the PLLcircuit 21.

A description will be provided by using a timing chart shown in FIG. 15as to the processing flow in the configuration.

In descending vertical order of the signals and processes of FIG. 15, VDindicates a vertical synchronizing signal for configuring the movingimage, reference, character b indicates a reduction processing forcreating an image configuring the moving image and buffering operationtiming of the reduced data in the reduction processing means 17 and thebuffer 18, reference character a indicates timing for storing andstoring the data in the storing means 16 for storing the still image bythe entire pixel memories 6 and 8, and reference character c indicates atiming for performing image processing in the image processing portion7.

(Moving Image Sensing Period)

First, in a data flow upon sensing the moving image normally, the periodup to an arrow portion below the point described as the still imagecapture in FIG. 15 represents a moving image sensing period, and thetiming between a VD signal and a next VD signal represents a unit frameperiod configuring the moving image. The timing generator 4 iscontrolled to start reading out the imaging data corresponding to allthe pixels with the image sensing device 1 ready for reading out all thepixels.

Next, according to timing of the timing generator 4, the data digitizedby the AD conversion portion 3 undergoes image reduction by thereduction processing means 17 and temporary buffering of the reducedimage by the buffer 18 in the order of the pieces of the data read outso as to configure the processed image for the moving image.

The image data for the moving image temporarily buffered in the buffer18 has its moving image selected by the selecting means 19. After theelapse of a predetermined time from the start of moving image bufferingin the portion of FIG. 3 indicated by reference letter b, the movingimage processing shown in the portion of FIG. 3 indicated by referenceletter c is sequentially performed by the image processing means 7. Inthis case, the predetermined time is longer than the predetermined timeaccording to the first embodiment, and so the moving image processingstarting time is accordingly delayed.

The moving image processing is set in a time period not to overtake thedata buffered in the buffer 18 so as to perform the moving imageprocessing at high speed to an extent of keeping it within a movingimage frame. As the high-speed clock of the PLL circuit 21 has a highermultiplication rate than that of the PLL circuit 21 circuit block of thefirst embodiment, the moving image can be processed at a higher speed.The processing contents and signal flow of the other blocks indicatedare the same as the contents according to the first embodiment.

(Capture Still Image)

According to this embodiment, however, the processing speed of themoving image is increased as indicated above. Therefore, thenon-processing periods are extended when having frame pixel dataconfigured as the still image divided for each of the areas andimage-processed in the non-processing periods. As the non-processingperiods are extended, divided amounts become smaller when divided intothe blanks of FIG. 15. FIG. 15 shows an example in which the processingis possible in 2 VD periods.

This embodiment is described a configuration example of the firstembodiment. However, the same is applicable to the second, third,fourth, fifth and sixth embodiments, where the moving image processingafter the moving image buffering is sped up so as to reduce the periodsdivided by the still image processing. It is also arranged to process alarge number of frames capable of continuous sensing on receiving astill image continuous sensing command.

As the first effect of realizing this embodiment as described above, itis possible, when capturing the still image of a larger size than themoving image while taking the moving image, to deliver it on the networkor record it without lowering the frame rate of the moving image at all.

As the second effect, it is possible to reduce the number of divisionsof the still image and reduce the still image processing period.

As the third effect, it is possible to increase the number of frames oflarge-screen still image continuous sensing.

As the fourth effect, it is possible to reduce the amount of the entirepixel memory 6 if the large-screen still image continuous sensing iswithin a prescribed number of frames.

[Embodiment of Moving Image and Still Image Processing Means]

(Example of the Structure)

FIG. 16 is a configuration block diagram showing a hardware structure ofa moving image and still image processing means of the presentinvention. In FIG. 16, various processing blocks are realized byexecuting respective programs, however, some of the processing blockscan be realized by hardware logic.

In FIG. 16, reference numeral 160 indicates a CPU controlling the movingimage and still image processing means 5 or 24. This CPU may be used forprocessing of the control means 9 or the other blocks.

Reference numeral 161 indicates a program memory for storing programs.The program memory 161 includes ROM, RAM, a disc and so on in whichpreferable programs are stored. In FIG. 16, programs especially relatedto the first embodiment are shown. Reference numeral 161 a indicates anarea storing a system program which controls the whole apparatus byitself or cooperation with other CPUs. Reference numeral 161 b indicatesan area storing a control sequence program of the moving image and stillimage processing means 5 or 24, which is shown as a flowchart of thefirst embodiment in FIG. 18. Reference numeral 161 c indicates an areastoring an image reduction module which reduces the image data for themoving image in the reduction processing means 17. Reference numeral 161d indicates an area storing a still image storing module which storesstill image data in the memory 6 by the storing means 16. Referencenumeral 161 e indicates an area storing the other programs or modulescorresponding to the first or the other embodiments.

Reference numeral 162 indicates a data memory for storing data. The datamemory 162 includes ROM, RAM, a disc and so on in which preferableprograms are stored. In FIG. 16, data especially related to the firstembodiment are shown. Reference numeral 162 a indicates an area as amoving image buffer (for example, corresponding to the buffer 18 in FIG.2) buffering the image data reduced by the image reduction module 161 c.Reference numeral 162 b indicates an area storing a still image by thestill image storing module 161 d. Reference numeral 162 c indicates anarea storing a flag showing whether or not the still image capture hasbeen requested. Reference numeral 162 d indicates an area storing atable of a division number representing the number of portions intowhich the still image is divided, as its example is shown in FIG. 17.The division number may have been already set, but not selected from thetable 162 d. Reference numeral 162 e indicates an area storing a stillimage output counter showing whether or not the divided whole stillimage data have been processed. Reference numeral 162 f indicates anarea storing a table of delay time from moving image getting time tostart time of moving image processing, as its example is shown in FIG.17. The delay time may have been already set, but not selected from thetable 162 f. Reference numeral g indicates an area storing the otherdata not shown in FIG. 16.

FIG. 17 is a diagram showing a table 162 d for a number representing thenumber of portions in which a still image is divided, and a table 162 ffor the delay time until the starting of a moving image processing ofthe present invention. The tables 162 d and 162 f can be combined tomake one table.

For example, the table 162 d is formed so that the division number 175is determined to correspond to the CD cycle time 171, the delay time ofthe moving image processing 172, the moving image size 177, the stillimage size 174 and so on. The table 162 f is formed so that the delaytime 179 is determined to correspond to the CD cycle time 176, themoving image size 177, the moving image processing rate (or clock rate)and so on.

(Control Sequence Program of the Moving Image and Still Image ProcessingMeans)

FIG. 18 is a flowchart showing a sequence of steps at the moving imageand still image processing means of the present invention. FIG. 18 isdescribed along a sequence of the first embodiment.

Step S181 is a step for synchronizing the image processing with the VD(vertical synchronizing signal). When detecting VD in step S181, CPU 160determines whether there is a still image capture operation between thelast VD and the current VD in step S182. Since, at the first VD in FIG.3, there is no still image capture operation, the CPU 160 determineswhether or not the still image capture flag 162 c is set in step S183.Since, at the first VD in FIG. 3, the flag 162 c is not set, a flow isadvanced to step S184. In step S184, the CPU 160 instructs the reductionand buffering of the moving image data. In step S184 of the second andthird embodiment, the buffering process is deleted. After waiting untilthe delay time has elapsed in step S185, the CPU 160 outputs the reducedmoving data to the image processing means 7 in step S186. In the thirdembodiment, the output destination is the encode processing portion 10.

When detecting the next VD (the second VD in FIG. 3), the CPU 160advances from S182 to S187 since there has been a still image captureoperation after the last VD. In step S187, the CPU 160 sets the stillimage capture flag 162 c. The CPU 160 instructs the storing of the stillimage data in the memory 6 in step S189 as well the reduction andbuffering of the moving image data in step S188 in parallel. In stepS190, the CPU 160 sets the division number n of the still image data inthe counter 162 e. After waiting until the delay time has elapsed instep S191, the CPU 160 outputs the reduced moving data to the imageprocessing means 7 in step S192.

When detecting the next VD (the third VD in FIG. 3), the CPU 160advances from S183 to S193 since the still image capture flag 162 c hasbeen set in step S187. In step S193, the CPU 160 instructs the reductionand buffering of the moving image data. In step S194, the CPU 160outputs each divided still image data to the image processing means 7.In step S195, the CPU 160 decrements the counter 162 e. In step S196,the CPU 160 determines whether or not the counter 162 e=0. Since thecounter is not 0, after waiting until the delay time has elapsed in stepS198, the CPU 160 outputs the reduced moving data to the imageprocessing means 7 in step S199. Until the counter 162 e=0, the outputto the image processing means 7 (or the encode processing portion 10 inthe third embodiment) of the divided still image data and the reducedmoving image data steps S193-199 are repeated.

For example in FIG. 3, when 4 divided still image data has beenoutputted, the counter 162 e=0. In this case, the CPU 160 resets thestill image capture flag 162 c in step S197. Therefore, from the nextVD, the CPU 160 executes steps S184-S186 to output only the reducedmoving image data to the image processing means 7.

It is apparent that flowcharts of the other embodiments can be also madeby modifying FIG. 18 or adding some steps to FIG. 18.

Other Embodiments of the Present Invention

Though the first to seventh embodiments are described independently inthis specification, they can be combined. Further, the intermediateembodiments which are made by modifying the typical embodimentsdescribed above or by adding/deleting some elements to/from thoseembodiments are included in the present invention.

The means configuring the image sensing apparatus and the steps of theimaging method of the embodiments of the aforementioned presentinvention can be realized by operation of the programs stored in theRAM, ROM and the like of the computer. The present invention includesthe programs and a computer-readable storing medium having the programsrecorded therein.

The present invention can be implemented as a system, an apparatus, amethod, a program or a storing medium for instance as the embodimentthereof. To be more precise, it is applicable either to a systemconfigured by multiple devices or to an apparatus configured by onedevice.

The present invention also includes the case of accomplishing it bydirectly or remotely supplying the program (program corresponding to theflowcharts shown in FIGS. 1 to 15 of the embodiment) of software forrealizing the aforementioned functions of the embodiments to the systemor the apparatus so as to have the supplied program code read out andexecuted by the computer of the system or the apparatus.

Therefore, the present invention is also realized by the program codeitself installed on the computer for the sake of realizing the functionsand processes of the present invention on the computer. To be morespecific, the present invention also includes the computer programitself for realizing the functions and processes of the presentinvention.

In that case, it may be in the form of an object code, a programexecuted by an interpreter, script data supplied to an OS (operatingsystem) or the like if it has the functions of the program.

Examples of the storing medium for supplying the program code include afloppy (registered trademark) disk, a hard disk, an optical disk, amagneto-optical disk, an MO, a CD-ROM, a CD-R, a CD-RW, a magnetic tape,a nonvolatile memory card, a ROM and a DVD (a DVD-ROM, a DVD-R).

As for the method of supplying the program, it is also possible toconnect to a home page on the Internet by using a browser of a clientcomputer and download the computer program itself of the presentinvention or a compressed file including an auto-install function to thestoring medium such as a hard disk so as to supply the program.

It is also realizable by dividing the program code configuring theprogram of the present invention into multiple files and downloadingeach of the files from a different home page. To be more specific, thepresent invention also includes a WWW server for downloading programfiles for realizing the functions and processes of the present inventionon the computer to multiple users.

It is also possible to distribute the program of the present inventionto the users by encrypting and storing it in the storing medium such asa CD-ROM, have the users having cleared predetermined conditions,download key information for decrypting it from a home page via theInternet, and have the encrypted program executed by using the keyinformation and installed on the computer so as to realize it.

Further, it is to be understood that the functions of the abovedescribed embodiments may be accomplished not only by executing theprogram read out by a computer, but also by causing an OS (operatingsystem) or the like which operates on the computer to perform a part orall of the actual operations based on instructions of the program.

Further, it is to be understood that the functions of the abovedescribed embodiments may be accomplished by writing the program readout from the storing medium into a memory provided in an expansion boardinserted into a computer or in an expansion unit connected to thecomputer and then causing a CPU or the like provided in the expansionboard or the expansion unit to perform a part or all of the actualoperations based on instructions of the program.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

This application claims the benefit of Japanese Application No.2005-133185, filed on Apr. 28, 2005, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image sensing apparatus comprising: an inputting unit configured to input sensed image data; a reducing unit configured to reduce the sensed image data into reduced image data of a first density; a detecting unit configured to detect an instruction for sensing a still image during sensing of a moving image; a storing control unit configured to control a memory to store image data of a second density higher than the first density based on the sensed image data inputted after detection of the instruction for sensing the still image; and a processing unit configured to, in a case where the detecting unit detects the instruction for sensing a still image during sensing of a moving image, process a first moving image frame by using first reduced image data of the first density during a first time period, process a second moving image frame next to the first moving image frame by using second reduced image data of the first density during a second time period after the first time period, divide a still image frame into a plurality of parts by using the image data of the second density based on the sensed image data inputted after detection of the instruction for sensing the still image, process a first part of the divided still image frame during a third time period between the first and second time periods, process a third moving image frame next to the second moving image frame during a fourth time period after the second time period, and process a second part of the divided still image frame during a time period between the second time period and the fourth time period, wherein during the third time period, the processing unit does not process the first or second moving image frames regardless of whether or not the detecting unit detects the instruction for sensing the still image during sensing of the moving image.
 2. The apparatus according to claim 1, wherein the processing unit processes a part of the still image frame during the time period during which the second moving image frame is stored in the memory in a case where the detecting unit detected the instruction for sensing the still image of the second density during sensing of the moving image frame of the first density.
 3. The apparatus according to claim 1, wherein the processing unit divides the still image frame into the plurality of parts based on at least an image size of the still image frame, an image size of the moving image frame, and a cycle time of a vertical synchronization signal of the moving image frame.
 4. The apparatus according to claim 2, wherein the processing unit has a table for setting a delayed time for which the second moving image frame is stored in the memory, and the delayed time is set based on at least a cycle time of a vertical synchronizing signal, the first density, and the processing rate of the processing unit.
 5. The apparatus according to claim 1, further comprising an image sensing unit having a moving image channel reading out one of the moving image frames within a cycle time of a vertical synchronizing signal, and a still image channel reading out the still image frame over plural cycle times of the vertical synchronizing signal, and first processing of one of the moving image frames by processing unit after detecting of the instruction for sensing the still image is performed based on one of the moving image frames acquired by reducing the still image frame composed of plural still image data obtained through the still image channel.
 6. The apparatus according to claim 5, wherein when the image sensing unit needs exposure time to obtain still image data of the still image frame through the still image channel, the processing unit uses a first cycle time of the vertical synchronizing signal as the exposure time, the next plural successive cycle time of the vertical synchronizing signal for obtaining image data through the still image channel, and the further next cycle time of the vertical synchronizing signal for composing the obtained still image data and reducing the composed still image data.
 7. A method of controlling an image sensing apparatus comprising: an inputting step of inputting sensed image data; a reducing step of reducing the sensed image data into reduced image data of a first density; a detection step of detecting an instruction for sensing a still image during sensing of a moving image; a storing controlling step of controlling a memory to store image data of a second density higher than the first density based on the sensed image data inputted after detection of the instruction for sensing the still image; and a processing step of, in a case where the detecting step detects the instruction for sensing a still image during sensing of a moving image, processing a first moving image frame by using first reduced image data of the first density during a first time period, processing a second moving image frame next to the first moving image frame by using second reduced image data of the first density during a second time period, dividing a still image frame into a plurality of parts by using the image data of the second density based on the sensed image data inputted after detection of the instruction for sensing the still image during a third time period, processing a first part of the divided still image frame during a third time period between the first and second time periods, processing a third moving image frame next to the second moving image frame during a fourth time period after the second time period, and processing a second part of the divided still image frame during a time period between the second time period and the fourth time period, wherein during the third time period, the processing step does not process the first or second moving image frames regardless of whether or not the detecting step detects the instruction for sensing the still image during sensing of the moving image.
 8. The method according to claim 7, wherein the processing step processes a part of the still image frame during the time period during which the second moving image frame is stored in the memory in a case where the instruction for sensing the still image of the second density was detected in the detection step during sensing of the moving image frame of the first density.
 9. The method according to claim 7, wherein the processing step divides the still image frame into the plurality of parts based on at least an image size of the still image frame, an image size of the moving image frame, and a cycle time of a vertical synchronization signal of the moving image frame.
 10. The method according to claim 8, wherein the delayed time during which the second moving image frame is stored in the memory is set based on at least a cycle time of a vertical synchronizing signal, the first density, and the processing rate of the processing step.
 11. The method according to claim 7, further comprising an image sensing step of sensing an image using an image sensing unit having a moving image channel reading out one of the moving image frames within a cycle time of a vertical synchronizing signal, and a still image channel reading out the still image frame over plural cycle times of the vertical synchronizing signal, and first processing of one of the moving image frames in the processing step after detecting of the instruction for sensing the still image is performed based on the first moving image frame acquired by reducing the still image frame composed of plural still image data obtained through the still image channel.
 12. The method according to claim 11, wherein in said processing step, when the image sensing unit requires data representing an exposure time to obtain the still image data of the still image frame through the still image channel, a first cycle time of the vertical synchronizing signal is used as the exposure time, a next plural successive cycle time of the vertical synchronizing signal is used for obtaining image data through the still image channel, and a further next cycle time of the vertical synchronizing signal is used for composing the obtained still image data and reducing the composed still image data.
 13. A non-transitory computer readable storage medium containing computer executable instructions for a computer, the computer executable instructions comprising: computer-executable instructions for performing a step of inputting sensed image data; computer-executable instructions for performing a step of reducing the sensed image data into reduced image data of a first density; computer-executable instructions for performing a step of detecting an instruction for sensing a still image during sensing of a moving image; computer-executable instructions for performing a step of controlling a memory to store image data of a second density higher than the first density based on the sensed image data inputted after detection of the instruction for sensing the still image; and computer-executable instructions for, in a case where the detecting step detects the instruction for the sensing a still image during sensing of a moving image, processing a first moving image frame by using first reduced image data of the first density during a first time period, processing a second moving image frame next to the first moving image frame by using second reduced image data of the first density during a second time period after the first time period, dividing a still image frame into a plurality of parts by using the image data of the second density based on the sensed image data inputted after detection of the instruction for sensing the still image, processing a first part of the divided still image frame during a third time period between the first and second time periods, processing a third moving image frame next to the second moving image frame during a fourth time period after the second time period, and processing a second part of the divided still image frame during a time period between the second time period and the fourth time period, wherein during the third time period, the processing step does not process the first or second moving image frames regardless of whether or not the detecting step detects the instruction for sensing the still image during sensing of the moving image. 